_files/ssr_blur.png)
Figure 10.1: Confirmation that Blur composition in each direction correctly blurs
This chapter describes the theory and implementation of Screen Space Reflection as an application of ImageEffect. When constructing a three-dimensional space, reflections and reflections are useful for expressing reality along with shadows. However, despite the simplicity of the phenomena we see in our daily lives, reflections and reflections are enormous calculations when trying to faithfully reproduce physical phenomena using ray tracing (described later) in the world of 3DCG. It is also an expression that requires quantity. Recently, Octan Renderer has become available in Unity, and when producing as a video work, it has become possible to produce quite photorealistic effects in Unity, but in real-time rendering it is still necessary to devise a pseudo reproduction. There is.
There are several techniques for expressing reflections with real-time rendering, but in this chapter we will introduce a technique called Screen Space Reflection (SSR) that belongs to the post-effects.
As for the structure of this chapter, we will first explain the blur processing used in the sample program in advance as a shoulder break-in for post effects. After that, I will explain SSR while breaking it down into the smallest possible processing units.
In addition, the sample of this chapter is in "SSR" of
https://github.com/IndieVisualLab/UnityGraphicsProgramming2
.
In this section, we will explain the blur processing. If you include anti-aliasing, you need to understand the procedure that blurring is very complicated, but this time it is a basic process because it is a shoulder break-in. The basis of blur processing is to homogenize the color of texels by multiplying each texel (pixels after rasterization * 4 ) of the image to be processed by a matrix that refers to the texels around it. I will continue. The matrix that references the texels around this is called the kernel. The kernel is a matrix that determines the proportion of texel colors mixed.
Gaussian blur is the most commonly used blur treatment. As the name implies, this refers to the process of using a Gaussian distribution in the kernel. Read the Gaussian Blur implementation diagonally to get a feel for how it works in post-effects.
The Gaussian kernel mixes the brightness around the pixel to be processed at a rate that follows a Gaussian distribution. By doing this, it is possible to suppress the blurring of the contour part where the brightness changes non-linearly.
As a review of mathematics, the Gaussian distribution can be expressed by the following formula.
G\left( x\right) =\dfrac {1}{\sqrt {2 \pi \sigma ^{2}}}\exp \left( -\dfrac {x^{2}}{2\sigma ^{2}}\right)
Since the Gaussian distribution can be approximated to the binomial distribution here, the Gaussian distribution can be substituted by the combination of weighting according to the binomial distribution as shown below (see footnote * 2 for the approximation of the Gaussian and binomial distributions ).
GaussianBlur.shader
float4 x_blur (v2f i) : SV_Target
{
float weight [5] = { 0.2270270, 0.1945945, 0.1216216, 0.0540540, 0.0162162 };
float offset [5] = { 0.0, 1.0, 2.0, 3.0, 4.0 };
float2 size = _MainTex_TexelSize;
fixed4 col = tex2D(_MainTex, i.uv) * weight[0];
for(int j=1; j<5; j++)
{
col += tex2D(_MainTex, i.uv + float2(offset[j], 0) * size) * weight[j];
col += tex2D(_MainTex, i.uv - float2(offset[j], 0) * size) * weight[j];
}
return col;
}
The above code is only in the x direction, but the processing is almost the same in the y direction. Here, the blur in the x and y directions is divided into two directions, and the number of brightness acquisitions is reduced from n * n = n ^ 2 times to n * 2 + 1 = 2n + 1 times. Because you can.
Figure 10.1: Confirmation that Blur composition in each direction correctly blurs
On the script side, OnRenderImageBlit alternately between src and temporary RenderTexture in each direction of xy, and finally Blit from src to dst and output. On MacOS, Blitt was possible only with src, but on Windows, the result was not output, so RenderTexture.GetTemporaryI am using. (For OnRenderImage and Blit, refer to the introduction to ImageEffect in the previous chapter.)
GaussianBlur.cs
void OnRenderImage (RenderTexture src, RenderTexture dst)
{
var rt = RenderTexture.GetTemporary(src.width, src.height, 0, src.format);
for (int i = 0; i < blurNum; i++)
{
Graphics.Blit(src, rt, mat, 0);
Graphics.Blit(rt, src, mat, 1);
}
Graphics.Blit(src, dst);
RenderTexture.ReleaseTemporary(rt);
}
This is the end of the explanation of Gaussian blur. Now that you have a sense of how post-effects are performed, I will explain SSR from the next section.
SSR is a technique that attempts to reproduce reflections and reflections within the range of post effects. All that is required for SSR is the image itself taken by the camera, the depth buffer in which the depth information is written, and the normal buffer in which the normal information is written. Depth buffer and normal buffer are collectively called G-buffer and are indispensable for Deferred rendering such as SSR. (For Deferred Rendering, there is a great explanation in the introduction to ImageEffect in the previous chapter, so please refer to that.)
This is a premise when reading this section, but in this section, we will proceed with the explanation on the premise of basic knowledge about ray tracing. Ray tracing is a big theme that I can write another chapter even at the introductory level, so unfortunately I will omit the explanation here. However, if you do not understand what ray tracing is, you can not understand the following contents, so if you do not understand it, there is a good introduction book "Ray Tracing in One Weekend" * 3 by Peter Shirley, so it is recommended that you read that first. I will.
In addition, kode80's "Screen Space Reflections in Unity 5 * 6 " is famous as a commentary text for the Unity implementation of SSR . Also, in Japanese text, there is "I tried to implement Screen Space Reflection in Unity * 8 ". In this section, what is explained in the above text is simplified as much as possible, and explanation of branch and leaf techniques is omitted. If you read the source code and find any questions, try to hit them.
The basic idea of SSR is to use ray tracing techniques to simulate the relationship between a camera, a reflective surface, and an object (light source).
Unlike ordinary optics, SSR reproduces reflections on the reflecting surface by fetching the color on the reflecting surface after identifying the light source by calculating back from the path of light incident on the camera.
_files/ssr_idea.png)
Figure 10.2: Differences between real-life optics and SSR light thinking
SSR does this for each pixel of the camera.
The outline of the process can be summarized as follows.
The procedure is difficult to explain in the figure, but it is complicated to explain in words. Let's disassemble it.
First, pass the matrix for converting the screen coordinate system and the world coordinate system to the shader. _ViewProjIs the transformation matrix from the world coordinate system to the screen coordinate system, and _InvViewProjis the inverse matrix.
SSR.cs
void OnRenderImage (RenderTexture src, RenderTexture dst)
{
....
// world <-> screen matrix
var view = cam.worldToCameraMatrix;
var proj = GL.GetGPUProjectionMatrix(cam.projectionMatrix, false);
var viewProj = proj * view;
mat.SetMatrix("_ViewProj", viewProj);
mat.SetMatrix("_InvViewProj", viewProj.inverse);
....
}
Now, using the transformation matrix passed, the normal vector and the reflection vector can be obtained. Let's take a look at the processing of the corresponding shader.
SSR.shader
float4 reflection (v2f i) : SV_Target
{
float2 uv = i.screen.xy / i.screen.w;
float depth = SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, uv);
...
float2 screenpos = 2.0 * uv - 1.0;
float4 pos = mul(_InvViewProj, float4(screenpos, depth, 1.0));
pos /= pos.w;
float3 camDir = normalize(pos - _WorldSpaceCameraPos);
float3 normal = tex2D(_CameraGBufferTexture2, uv) * 2.0 - 1.0;
float3 refDir = reflect(camDir, normal);
....
if (_ViewMode == 1) col = float4((normal.xyz * 0.5 + 0.5), 1);
if (_ViewMode == 2) col = float4((refDir.xyz * 0.5 + 0.5), 1);
....
return col;
}
First, the depth of the corresponding pixel is _CameraDepthTexturewritten in, and this is used. Next, from the position information and seismic intensity information on the screen, the position of the polygon in the corresponding pixel in the world coordinate system can be found, so poshold it in. posThen, _WorldSpaceCameraPossince the vector toward the camera is known, the reflection vector can be known from this and the normal information.
From the script attached to the main camera, you can see where the normal and reflection vectors are facing. Since each vector is standardized between -1 and 1, color information with a value less than or equal to 0 is not displayed. When the x-axis component is large, the vector is displayed in reddish, when the y-axis component is large, it is displayed in greenish, and when the z-axis component is large, it is displayed in bluish. Please set ViewMode to Normalor Reflectionand check.
Now let's look at the process of performing ray tracing.
SSR.shader
float4 reflection(v2f i) : SV_Target
{
...
[loop]
for (int n = 1; n <= _MaxLoop; n++)
{
float3 step = refDir * _RayLenCoeff * (lod + 1);
ray += step * (1 + rand(uv + _Time.x) * (1 - smooth));
float4 rayScreen = mul (_ViewProj, float4 (ray, 1.0));
float2 rayUV = rayScreen.xy / rayScreen.w * 0.5 + 0.5;
float rayDepth = ComputeDepth(rayScreen);
float worldDepth = (lod == 0)?
SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, rayUV) :
tex2Dlod (_CameraDepthMipmap, float4 (rayUV, 0, lod))
+ _BaseRaise * lod;
...
if(rayDepth < worldDepth)
{
....
return outcol;
}
}
}
Variables related to the processing explained later are also mixed, but please read it without worrying about it. Inside the loop, first stretch the ray by a step and then put it back in the screen coordinate system. We will compare the depth of the ray in the screen coordinate system with the depth written to the depth buffer and return the color if the ray is deeper. (1.0 when the depth is the closest, because the smaller the further away, rayDepthbut worldDepthis smaller than the will to a determination that Ray is in the back.)
Also, if the number of loops is undecided, HLSL will throw an error, so if you want to pass the number of loops from the script [loop], you need to write the attribute at the beginning.
The skeleton of ray tracing is now complete. The basic processing is not so difficult once you have an image. However, in order to reproduce beautiful reflections, it is necessary to add some processing from now on. Are the following four points that need to be improved?
For post-effects, including antialiasing, techniques for efficient processing are rather essential. Now that you understand the gist of the process, let's look at the technique for establishing SSR as a video.
Below, we will explain how to improve processing efficiency by using Mipmap, referring to the article * 7 of Chalmers University of Technology . (See footnote for what Mipmap is * 9 ) Ray tracing basically determines the step width of the ray and gradually advances the ray, but by using Mipmap, the step of the ray until the intersection with the object is judged. The width can be variable. By doing this, you will be able to fly rays far away even with a limited number of loops, and processing efficiency will also increase.
We have prepared a demoscene that uses Mipmap from RenderTexture, so let's check it from there.
Mipmap.cs
public class Mipmap : MonoBehaviour
{
Material mat;
RenderTexture rt;
[SerializeField] Shader shader;
[SerializeField] int lod;
void OnEnable()
{
mat = new Material(shader);
rt = new RenderTexture(Screen.width, Screen.height, 24);
rt.useMipMap = true;
}
void OnDisable()
{
Destroy(mat);
rt.Release();
}
void OnRenderImage (RenderTexture src, RenderTexture dst)
{
mat.SetInt("_LOD", lod);
Graphics.Blit(src, rt);
Graphics.Blit(rt, dst, mat);
}
}
Since mipmap cannot be set for ready-made RenderTexture, here, srcafter creating a new RenderTexture and copying it, processing is added.
Mipmap.shader
sampler2D _MainTex;
float4 _MainTex_ST;
int _LOD;
....
fixed4 frag (v2f i) : SV_Target
{
return tex2Dlod(_MainTex, float4(i.uv, 0, _LOD));
}
tex2Dlod(_MainTex, float4(i.uv, 0, _LOD))You can get the Mipmap according to the LOD with.
If you raise the LOD from the script attached to the camera on the scene, you can see that the image becomes grainy.
_files/ssr_mipmap_demo.png)
Figure 10.3: Comparison of increased LOD and Mipmap image quality
Now that you have confirmed how to use Mipmap, let's see how Mipmap is used in the SSR scene.
SSR.shader
[loop]
for (int n = 1; n <= _MaxLoop; n++)
{
float3 step = refDir * _RayLenCoeff * (lod + 1);
ray += step;
....
if(rayDepth < worldDepth)
{
if(lod == 0)
{
if (rayDepth + _Thickness > worldDepth)
{
float sign = -1.0;
for (int m = 1; m <= 8; ++m)
{
ray += sign * pow(0.5, m) * step;
rayScreen = mul (_ViewProj, float4 (ray, 1.0));
rayUV = rayScreen.xy / rayScreen.w * 0.5 + 0.5;
rayDepth = ComputeDepth(rayScreen);
worldDepth = SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, rayUV);
sign = (rayDepth < worldDepth) ? -1 : 1;
}
refcol = tex2D(_MainTex, rayUV);
}
break;
}
else
{
ray -= step;
lod--;
}
}
else if(n <= _MaxLOD)
{
lod ++;
}
calcTimes = n;
}
if (_ViewMode == 3) return float4(1, 1, 1, 1) * calc / _MaxLoop;
....
I will proceed with the explanation using the figure in the article of Chalmers.
_files/ssr_mipmap_logic.png)
Figure 10.4: Calculation method using Mipmap
As shown in the figure, the LOD is raised while carefully judging the intersection for the first few times. As long as there is no intersection with other meshes, we will proceed with large steps. If there is an intersection, Unity's MipMap will roughen the pixels while taking the average value, so unlike the case of the article, the ray may go too far. Therefore, move back by one unit step and advance the ray again with one smaller LOD. Finally, by making an intersection judgment on the image with LOD = 0, the moving distance of the ray can be extended and the processing can be made more efficient.
From the script attached to the main camera, you can see how much the amount of calculation changes when you raise the LOD. The larger the amount of calculation, the whiter it looks, and the smaller the amount of calculation, the darker it looks. Set ViewMode and CalcCountchange the LOD to check the change in the amount of calculation.
_files/ssr_lod.png)
Figure 10.5: Difference in computational complexity due to changes in LOD (closer to black, smaller computational complexity)
Let's see how to improve the accuracy near the intersection by binary tree search. Check from the code immediately.
SSR.shader
if (lod == 0)
{
if (rayDepth + _Thickness > worldDepth)
{
float sign = -1.0;
for (int m = 1; m <= 8; ++m)
{
ray += sign * pow(0.5, m) * step;
rayScreen = mul (_ViewProj, float4 (ray, 1.0));
rayUV = rayScreen.xy / rayScreen.w * 0.5 + 0.5;
rayDepth = ComputeDepth(rayScreen);
worldDepth = SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, rayUV);
sign = (rayDepth < worldDepth) ? -1 : 1;
}
refcol = tex2D(_MainTex, rayUV);
}
break;
}
Immediately after the intersection, it is behind the intersected object, so first retract the ray. After that, while checking the context of the ray and the mesh, change the direction of travel of the ray either forward or backward. At the same time, by shortening the step width of the ray, it is possible to identify the intersection with the mesh with less error.
The methods so far have not taken into account the differences in the materials of the objects in the screen. Therefore, there is a problem that all objects are reflected to the same extent. Therefore, use G-buffer again. _CameraGBufferTexture1.wSince the smoothness of the material is stored in, use this.
SSR.shader
if (_ViewMode == 8) return float4(1, 1, 1, 1) * tex2D(_CameraGBufferTexture1, uv).w; .... return (col * (1 - smooth) + refcol * smooth) * _ReflectionRate + col * (1 - _ReflectionRate);
If you change the smoothness value of the material attached to an object in the scene, you can see that only that object changes the degree of reflection. SmoothnessYou can also list the smoothness in the scene by setting the ViewMode of the script attached to the main camera . The whitish, the greater the smoothness.
This is the part using the Gaussian blur explained in the first section. If the step width of the ray is not small enough, you may not be able to get the reflection well even if you perform a binary tree search. If the step width of the ray is reduced, the total length of the ray will be shortened and the amount of calculation will increase, so it is not enough to just reduce the step width, but it should be kept to an appropriate size. The part where the reflection could not be obtained well is blurred to make it look like it.
SSR.shader
float4 xblur(v2f i) : SV_Target
{
float2 uv = i.screen.xy / i.screen.w;
float2 size = _ReflectionTexture_TexelSize;
float smooth = tex2D(_CameraGBufferTexture1, uv).w;
// compare depth
float depth = SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, uv);
float depthR =
SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, uv + float2(1, 0) * size);
float depthL =
SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, uv - float2(1, 0) * size);
if (depth <= 0) return tex2D(_ReflectionTexture, uv);
float weight[5] = { 0.2270270, 0.1945945, 0.1216216, 0.0540540, 0.0162162 };
float offset[5] = { 0.0, 1.0, 2.0, 3.0, 4.0 };
float4 originalColor = tex2D(_ReflectionTexture, uv);
float4 blurredColor = tex2D(_ReflectionTexture, uv) * weight[0];
for (int j = 1; j < 5; ++j)
{
blurredColor
+= tex2D(_ReflectionTexture, uv + float2(offset[j], 0) * size)
* weight[j];
blurredColor
+= tex2D(_ReflectionTexture, uv - float2(offset[j], 0) * size)
* weight[j];
}
float4 o = (abs(depthR - depthL) > _BlurThreshold) ? originalColor
: blurredColor * smooth + originalColor * (1 - smooth);
return o;
}
Again, from the reason described above xblurand yblurhave divided the processing out. Also, since we only want to blur the contours within the same reflective surface, we try not to blur the contours. If the difference between the left and right depths is large, it is judged to be the contour part. (Then yblurevaluates the difference between the top and bottom.)
The result of adding the processing up to this point is as follows.
_files/ssr_result.png)
Figure 10.6: Results
As a bonus, I will introduce a technique that makes it look as if a non-existent object is reflected using two cameras, a main camera and a sub camera.
SSRMainCamera.shader
float4 reflection(v2f i) : SV_Target
{
....
for (int n = 1; n <= 100; ++n)
{
float3 ray = n * step;
float3 rayPos = pos + ray;
float4 vpPos = mul (_ViewProj, float4 (rayPos, 1.0));
float2 rayUv = vpPos.xy / vpPos.w * 0.5 + 0.5;
float rayDepth = vpPos.z / vpPos.w;
float subCameraDepth = SAMPLE_DEPTH_TEXTURE(_SubCameraDepthTex, rayUv);
if (rayDepth < subCameraDepth && rayDepth + thickness > subCameraDepth)
{
float sign = -1.0;
for (int m = 1; m <= 4; ++m)
{
rayPos += sign * pow(0.5, m) * step;
vpPos = mul (_ViewProj, float4 (rayPos, 1.0));
rayUv = vpPos.xy / vpPos.w * 0.5 + 0.5;
rayDepth = vpPos.z / vpPos.w;
subCameraDepth = SAMPLE_DEPTH_TEXTURE(_SubCameraDepthTex, rayUv);
sign = rayDepth - subCameraDepth < 0 ? -1 : 1;
}
col = tex2D (_SubCameraMainTex, rayUv);
}
}
return col * smooth + tex2D(_MainTex, uv) * (1 - smooth);
}
It is made simple with as little extra processing as possible. The point is that SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, uv)it SAMPLE_DEPTH_TEXTURE(_SubCameraDepthTex, rayUv)is used instead of for depth evaluation, and the object information to be referenced _SubCameraMainTexis also obtained from. _CameraDepthTexture, _SubCameraDepthTexIs set as a global texture from the sub camera.
The downside is that each camera casts shadows on objects that shouldn't be visible. It may not be very practical, but it's a little interesting effect.
_files/ssr_twocam.png)
Figure 10.7: Method using two cameras
This is the end of the explanation of SSR.
Since SSR is a technique that requires a large amount of processing capacity, it is not realistic to reflect objects in all positions cleanly. Therefore, the point is to improve the appearance of the reflection of the object of interest and to make the trivial reflection look like it with less processing. In addition, the screen size to be rendered is directly linked to the amount of calculation, so it is important to search for the points that will be established as an image while considering the expected screen size and GPU performance. Check the role and trade-offs of each parameter by adjusting the parameters while moving the objects in the scene.
In addition, the Mipmap, binary tree search, how to use the camera buffer, and many other detailed techniques mentioned above can be applied not only to SSR but also to various places. I would be happy if there is some content that is helpful to the readers.
[*1] http://rastergrid.com/blog/2010/09/efficient-gaussian-blur-with-linear-sampling/
[*2] https://ja.wikipedia.org/wiki/%E4%BA%8C%E9%A0%85%E5%88%86%E5%B8%83
[*3] https://www.amazon.co.jp/gp/product/B01B5AODD8
[*4] https://msdn.microsoft.com/ja-jp/library/bb219690(v=vs.85).aspx
[*5] https://www.sciencelearn.org.nz/resources/48-reflection-of-light
[*6] http://www.kode80.com/blog/2015/03/11/screen-space-reflections-in-unity-5/
[*7] http://www.cse.chalmers.se/edu/year/2017/course/TDA361/ Advanced%20Computer%20Graphics/Screen-space%20reflections.pdf
[*8] http://tips.hecomi.com/entry/2016/04/04/022550
[*9] https://answers.unity.com/questions/441984/what-is-mip-maps-pictures.html
[*10] https://docs.unity3d.com/Manual/RenderTech-DeferredShading.html